SherbyNet - /page/wikibook/entry.mu

<!DOCTYPE html>

<title>Normalizing constant</title>

</head>

<span id="openzim-page-title" class="mw-page-title-main"><span class="mw-page-title-main">Normalizing constant</span></span>

</h1>

</header>

</p>

/* start https://en.wikipedia.org/ */

.mw-parser-output .hatnote{font-style:italic}.mw-parser-output div.hatnote{padding-left:1.6em;margin-bottom:0.5em}.mw-parser-output .hatnote i{font-style:normal}.mw-parser-output .hatnote+link+.hatnote{margin-top:-0.5em}@media print{body.ns-0 .mw-parser-output .hatnote{display:none!important}}

/* end https://en.wikipedia.org/ */

</style><div role="note" class="hatnote navigation-not-searchable">Not to be confused with <a href="Proportionality_factor" class="mw-redirect" title="Proportionality factor">Proportionality factor</a>.</div>

<p>In <a href="Probability_theory" title="Probability theory">probability theory</a>, a <b>normalizing constant</b> or <b>normalizing factor</b> is used to reduce any probability function to a probability density function with total probability of one.

</p><p>For example, a Gaussian function can be normalized into a probability density function, which gives the standard normal distribution. In Bayes' theorem, a normalizing constant is used to ensure that the sum of all possible hypotheses equals 1. Other uses of normalizing constants include making the value of a Legendre polynomial at 1 and in the orthogonality of orthonormal functions.

</p><p>A similar concept has been used in areas other than probability, such as for polynomials.

</p>

<div class="mw-heading mw-heading2"><h2 id="Definition">Definition</h2></div>

<p>In <a href="Probability_theory" title="Probability theory">probability theory</a>, a <b>normalizing constant</b> is a constant by which an everywhere non-negative function must be multiplied so the area under its graph is 1, e.g., to make it a <a href="Probability_density_function" title="Probability density function">probability density function</a> or a <a href="Probability_mass_function" title="Probability mass function">probability mass function</a>.<sup id="cite_ref-1" class="reference"><a href="#cite_note-1"><span class="cite-bracket">[</span>1<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-2" class="reference"><a href="#cite_note-2"><span class="cite-bracket">[</span>2<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading2"><h2 id="Examples">Examples</h2></div>

<p>If we start from the simple <a href="Gaussian_function" title="Gaussian function">Gaussian function</a>

<msup>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle p(x)=e^{-x^{2}/2},\quad x\in (-\infty ,\infty )}</annotation>

</semantics>

</math></span></span>

we have the corresponding <a href="Gaussian_integral" title="Gaussian integral">Gaussian integral</a>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</msubsup>

<msup>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</msup>

<msqrt>

</msqrt>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \int _{-\infty }^{\infty }p(x)\,dx=\int _{-\infty }^{\infty }e^{-x^{2}/2}\,dx={\sqrt {2\pi \,}},}</annotation>

</semantics>

</math></span></span>

</p><p>Now if we use the latter's <a href="Reciprocal_value" class="mw-redirect" title="Reciprocal value">reciprocal value</a> as a normalizing constant for the former, defining a function <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \varphi (x)}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \varphi (x)}</annotation>

</semantics>

</math></span><img src="./4c4046f1f2de7df04bde418ba2bc4d3898ac2385.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:4.659ex; height:2.843ex;" alt="{\displaystyle \varphi (x)}" loading="lazy"></span> as

<mfrac>

<msqrt>

</msqrt>

</mfrac>

</mrow>

<mfrac>

<msqrt>

</msqrt>

</mfrac>

</mrow>

<msup>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \varphi (x)={\frac {1}{\sqrt {2\pi \,}}}p(x)={\frac {1}{\sqrt {2\pi \,}}}e^{-x^{2}/2}}</annotation>

</semantics>

</math></span></span>

so that its <a href="Integral_of_a_Gaussian_function" class="mw-redirect" title="Integral of a Gaussian function">integral</a> is unit

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</msubsup>

<mfrac>

<msqrt>

</msqrt>

</mfrac>

</mrow>

<msup>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \int _{-\infty }^{\infty }\varphi (x)\,dx=\int _{-\infty }^{\infty }{\frac {1}{\sqrt {2\pi \,}}}e^{-x^{2}/2}\,dx=1}</annotation>

</semantics>

</math></span></span>

then the function <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \varphi (x)}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \varphi (x)}</annotation>

</semantics>

</math></span><img src="./4c4046f1f2de7df04bde418ba2bc4d3898ac2385.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:4.659ex; height:2.843ex;" alt="{\displaystyle \varphi (x)}" loading="lazy"></span> is a probability density function.<sup id="cite_ref-3" class="reference"><a href="#cite_note-3"><span class="cite-bracket">[</span>3<span class="cite-bracket">]</span></a></sup> This is the density of the standard <a href="Normal_distribution" title="Normal distribution">normal distribution</a>. (<i>Standard</i>, in this case, means the <a href="Expected_value" title="Expected value">expected value</a> is 0 and the <a href="Variance" title="Variance">variance</a> is 1.)

</p><p>And constant <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\textstyle {\frac {1}{\sqrt {2\pi }}}}">

<mfrac>

<msqrt>

</msqrt>

</mfrac>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\textstyle {\frac {1}{\sqrt {2\pi }}}}</annotation>

</semantics>

</math></span><img src="./eccd2c28dc343be5631094e573191b2c17edd21d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.838ex; width:3.969ex; height:4.176ex;" alt="{\textstyle {\frac {1}{\sqrt {2\pi }}}}" loading="lazy"></span> is the <b>normalizing constant</b> of function <span class="mwe-math-element mwe-math-element-inline"><span class="mwe-math-mathml-inline mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle p(x)}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle p(x)}</annotation>

</semantics>

</math></span><img src="./8cb7afced134ef75572e5314a5d278c2d644f438.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; margin-left: -0.089ex; width:4.398ex; height:2.843ex;" alt="{\displaystyle p(x)}" loading="lazy"></span>.

</p><p>Similarly,

</mrow>

</mrow>

</munderover>

<mfrac>

<msup>

</mrow>

</msup>

<mrow>

</mrow>

</mfrac>

</mrow>

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \sum _{n=0}^{\infty }{\frac {\lambda ^{n}}{n!}}=e^{\lambda },}</annotation>

</semantics>

</math></span></span>

and consequently

<mfrac>

<mrow>

<msup>

</mrow>

</msup>

<msup>

</mrow>

</msup>

</mrow>

<mrow>

</mrow>

</mfrac>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle f(n)={\frac {\lambda ^{n}e^{-\lambda }}{n!}}}</annotation>

</semantics>

</math></span></span>

is a probability mass function on the set of all nonnegative integers.<sup id="cite_ref-4" class="reference"><a href="#cite_note-4"><span class="cite-bracket">[</span>4<span class="cite-bracket">]</span></a></sup> This is the probability mass function of the <a href="Poisson_distribution" title="Poisson distribution">Poisson distribution</a> with expected value λ.

</p><p>Note that if the probability density function is a function of various parameters, so too will be its normalizing constant. The parametrised normalizing constant for the <a href="Boltzmann_distribution" title="Boltzmann distribution">Boltzmann distribution</a> plays a central role in <a href="Statistical_mechanics" title="Statistical mechanics">statistical mechanics</a>. In that context, the normalizing constant is called the <a href="Partition_function_(statistical_mechanics)" title="Partition function (statistical mechanics)">partition function</a>.

</p>

<div class="mw-heading mw-heading2"><h2 id="Bayes'_theorem">Bayes' theorem</h2></div>

<p><a href="Bayes'_theorem" title="Bayes' theorem">Bayes' theorem</a> says that the posterior probability measure is proportional to the product of the prior probability measure and the <a href="Likelihood_function" title="Likelihood function">likelihood function</a>. <i>Proportional to</i> implies that one must multiply or divide by a normalizing constant to assign measure 1 to the whole space, i.e., to get a probability measure. In a simple discrete case we have

<msub>

</mrow>

</msub>

</mrow>

<mfrac>

<mrow>

</mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

<mrow>

</mrow>

</mfrac>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle P(H_{0}|D)={\frac {P(D|H_{0})P(H_{0})}{P(D)}}}</annotation>

</semantics>

</math></span></span>

where P(H<sub>0</sub>) is the prior probability that the hypothesis is true; P(D|H<sub>0</sub>) is the <a href="Conditional_probability" title="Conditional probability">conditional probability</a> of the data given that the hypothesis is true, but given that the data are known it is the <a href="Likelihood_function" title="Likelihood function">likelihood</a> of the hypothesis (or its parameters) given the data; P(H<sub>0</sub>|D) is the posterior probability that the hypothesis is true given the data. P(D) should be the probability of producing the data, but on its own is difficult to calculate, so an alternative way to describe this relationship is as one of proportionality:

<msub>

</mrow>

</msub>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle P(H_{0}|D)\propto P(D|H_{0})P(H_{0}).}</annotation>

</semantics>

</math></span></span>

Since P(H|D) is a probability, the sum over all possible (mutually exclusive) hypotheses should be 1, leading to the conclusion that

<msub>

</mrow>

</msub>

</mrow>

<mfrac>

<mrow>

</mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munder>

</mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mfrac>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle P(H_{0}|D)={\frac {P(D|H_{0})P(H_{0})}{\displaystyle \sum _{i}P(D|H_{i})P(H_{i})}}.}</annotation>

</semantics>

</math></span></span>

In this case, the <a href="Multiplicative_inverse" title="Multiplicative inverse">reciprocal</a> of the value

</mrow>

</munder>

</mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle P(D)=\sum _{i}P(D|H_{i})P(H_{i})\;}</annotation>

</semantics>

</math></span></span>

is the <i>normalizing constant</i>.<sup id="cite_ref-5" class="reference"><a href="#cite_note-5"><span class="cite-bracket">[</span>5<span class="cite-bracket">]</span></a></sup> It can be extended from countably many hypotheses to uncountably many by replacing the sum by an integral.

</p><p>For concreteness, there are many methods of estimating the normalizing constant for practical purposes. Methods include the bridge sampling technique, the naive Monte Carlo estimator, the generalized harmonic mean estimator, and importance sampling.<sup id="cite_ref-6" class="reference"><a href="#cite_note-6"><span class="cite-bracket">[</span>6<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading2"><h2 id="Non-probabilistic_uses">Non-probabilistic uses</h2></div>

<p>The <a href="Legendre_polynomials" title="Legendre polynomials">Legendre polynomials</a> are characterized by <a href="Orthogonality" title="Orthogonality">orthogonality</a> with respect to the uniform measure on the interval [−1, 1] and the fact that they are <b>normalized</b> so that their value at 1 is 1. The constant by which one multiplies a polynomial so its value at 1 is a normalizing constant.

</p><p><a href="Orthonormal" class="mw-redirect" title="Orthonormal">Orthonormal</a> functions are normalized such that <span class="mwe-math-element mwe-math-element-block"><span class="mwe-math-mathml-display mwe-math-mathml-a11y" style="display: none;"><math display="block" xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \langle f_{i},\,f_{j}\rangle =\,\delta _{i,j}}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \langle f_{i},\,f_{j}\rangle =\,\delta _{i,j}}</annotation>

</semantics>

</math></span></span> with respect to some inner product <span class="texhtml">⟨<i>f</i>, <i>g</i>⟩</span>.

</p><p>The constant <span class="texhtml">1/<span class="nowrap">√<span style="border-top:1px solid; padding:0 0.1em;">2</span></span></span> is used to establish the <a href="Hyperbolic_functions#Comparison_with_circular_functions" title="Hyperbolic functions">hyperbolic functions</a> cosh and sinh from the lengths of the adjacent and opposite sides of a <a href="Hyperbolic_sector#Hyperbolic_triangle" title="Hyperbolic sector">hyperbolic triangle</a>.

</p>

<ul><li><a href="Normalization_(statistics)" title="Normalization (statistics)">Normalization (statistics)</a></li></ul>

<div class="mw-heading mw-heading2"><h2 id="References">References</h2></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .reflist{margin-bottom:0.5em;list-style-type:decimal}@media screen{.mw-parser-output .reflist{font-size:90%}}.mw-parser-output .reflist .references{font-size:100%;margin-bottom:0;list-style-type:inherit}.mw-parser-output .reflist-columns-2{column-width:30em}.mw-parser-output .reflist-columns-3{column-width:25em}.mw-parser-output .reflist-columns{margin-top:0.3em}.mw-parser-output .reflist-columns ol{margin-top:0}.mw-parser-output .reflist-columns li{page-break-inside:avoid;break-inside:avoid-column}.mw-parser-output .reflist-upper-alpha{list-style-type:upper-alpha}.mw-parser-output .reflist-upper-roman{list-style-type:upper-roman}.mw-parser-output .reflist-lower-alpha{list-style-type:lower-alpha}.mw-parser-output .reflist-lower-greek{list-style-type:lower-greek}.mw-parser-output .reflist-lower-roman{list-style-type:lower-roman}

/* end https://en.wikipedia.org/ */

</style><div class="reflist">

<li id="cite_note-1"><span class="mw-cite-backlink"><b><a href="#cite_ref-1">^</a></b></span> <span class="reference-text"><a rel="nofollow" class="external text" href="http://www.math.uah.edu/stat/dist/Continuous.xhtml">Continuous Distributions</a> at Department of Mathematical Sciences: University of Alabama in Huntsville</span>

</li>

<li id="cite_note-2"><span class="mw-cite-backlink"><b><a href="#cite_ref-2">^</a></b></span> <span class="reference-text"><a href="#CITEREFFeller1968">Feller 1968</a>, p. 22</span>

</li>

<li id="cite_note-3"><span class="mw-cite-backlink"><b><a href="#cite_ref-3">^</a></b></span> <span class="reference-text"><a href="#CITEREFFeller1968">Feller 1968</a>, p. 174</span>

</li>

<li id="cite_note-4"><span class="mw-cite-backlink"><b><a href="#cite_ref-4">^</a></b></span> <span class="reference-text"><a href="#CITEREFFeller1968">Feller 1968</a>, p. 156</span>

</li>

<li id="cite_note-5"><span class="mw-cite-backlink"><b><a href="#cite_ref-5">^</a></b></span> <span class="reference-text"><a href="#CITEREFFeller1968">Feller 1968</a>, p. 124</span>

</li>

/* start https://en.wikipedia.org/ */

.mw-parser-output cite.citation{font-style:inherit;word-wrap:break-word}.mw-parser-output .citation q{quotes:"\"""\"""'""'"}.mw-parser-output .citation:target{background-color:rgba(0,127,255,0.133)}.mw-parser-output .id-lock-free.id-lock-free a{background:url("./mw/Lock-green.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-limited.id-lock-limited a,.mw-parser-output .id-lock-registration.id-lock-registration a{background:url("./mw/Lock-gray-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-subscription.id-lock-subscription a{background:url("./mw/Lock-red-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .cs1-ws-icon a{background:url("./mw/Wikisource-logo.svg")right 0.1em center/12px no-repeat}body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-free a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-limited a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-registration a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-subscription a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .cs1-ws-icon a{background-size:contain;padding:0 1em 0 0}.mw-parser-output .cs1-code{color:inherit;background:inherit;border:none;padding:inherit}.mw-parser-output .cs1-hidden-error{display:none;color:var(--color-error,#d33)}.mw-parser-output .cs1-visible-error{color:var(--color-error,#d33)}.mw-parser-output .cs1-maint{display:none;color:#085;margin-left:0.3em}.mw-parser-output .cs1-kern-left{padding-left:0.2em}.mw-parser-output .cs1-kern-right{padding-right:0.2em}.mw-parser-output .citation .mw-selflink{font-weight:inherit}@media screen{.mw-parser-output .cs1-format{font-size:95%}html.skin-theme-clientpref-night .mw-parser-output .cs1-maint{color:#18911f}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .cs1-maint{color:#18911f}}

/* end https://en.wikipedia.org/ */

</style><cite id="CITEREFGronau2020" class="citation web cs1">Gronau, Quentin (2020). <a rel="nofollow" class="external text" href="https://cran.r-project.org/web/packages/bridgesampling/vignettes/bridgesampling_paper.pdf">"bridgesampling: An R Package for Estimating Normalizing Constants"</a> <span class="cs1-format">(PDF)</span>. <i>The Comprehensive R Archive Network</i><span class="reference-accessdate">. Retrieved <span class="nowrap">September 11,</span> 2021</span>.</cite></span>

</li>

</ol></div></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .refbegin{margin-bottom:0.5em}.mw-parser-output .refbegin-hanging-indents>ul{margin-left:0}.mw-parser-output .refbegin-hanging-indents>ul>li{margin-left:0;padding-left:3.2em;text-indent:-3.2em}.mw-parser-output .refbegin-hanging-indents ul,.mw-parser-output .refbegin-hanging-indents ul li{list-style:none}@media(max-width:720px){.mw-parser-output .refbegin-hanging-indents>ul>li{padding-left:1.6em;text-indent:-1.6em}}.mw-parser-output .refbegin-columns{margin-top:0.3em}.mw-parser-output .refbegin-columns ul{margin-top:0}.mw-parser-output .refbegin-columns li{page-break-inside:avoid;break-inside:avoid-column}@media screen{.mw-parser-output .refbegin{font-size:90%}}

/* end https://en.wikipedia.org/ */

</style><div class="refbegin" style="">

<ul><li><cite id="CITEREFFeller1968" class="citation book cs1"><a href="William_Feller" title="William Feller">Feller, William</a> (1968). <i>An Introduction to Probability Theory and its Applications (volume I)</i>. John Wiley & Sons. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>0-471-25708-7</bdi>.</cite></li></ul>

</div></div><div><div class="zim-footer">

This article is issued from <a class="external text" title="Last edited on 2024-06-19" href="https://en.wikipedia.org/wiki/?title=Normalizing_constant&oldid=1229954511">Wikipedia</a>. The text is available under <a class="external text" href="https://creativecommons.org/licenses/by-sa/4.0/deed.en">Creative Commons Attribution-Share Alike 4.0</a> unless otherwise noted. Additional terms may apply for the media files.

</div>

</div></div>

</div>

</main>

</div>

</body></html>